System for analysis of motion

ABSTRACT

A computer system for real-time determination of a motion vector comprising an image processor to apply an image processing filter to a normalized frame element of a first image frame, yielding a filtered frame element having associated filtered pixel values, a pixel selector to select a first reference pixel from the filtered frame element having the highest value of the filtered pixel values, an optic flow module to determine a first optic flow applied at a location of the first reference pixel, a pattern matching module to perform pattern matching between the normalized pixel values of the normalized frame element and normalized frame elements of a second image frame, yielding a plurality of pattern matching scores, and a motion vector determiner to determine a motion vector based on a lowest one of the pattern matching scores. Related apparatus and methods are also described.

FIELD OF THE INVENTION

The present invention relates to efficient computation of a motionvector of an object imaged in a stream of video image frames.

BACKGROUND OF THE INVENTION

Real time computation of a motion vector of an object imaged in a streamof video image frames is typically performed using specialized hardware,or using software requiring intensive processing.

The prior art of motion field estimation is largely dedicated tohardware for specific applications, such as fluid metrics and targettracking. The applications typically require either expensive equipmentor precise estimation. There is currently no product that implements a“lightweight” real time implementation of optical flow concepts.

A system which uses a connected camera and motion detection in order toprovide interactive games, is, for example, the Sony Playstation®. TheSony product is named EyeToy™, and other variations on the name EyeToy™such as, EyeToy: Play 2™. An example reference to Sony EyeToy™ can befound on the World Wide Web, at:www.us.playstation.com/pressreleases.aspx?id=283.

The following references are also believed to represent the state of theart:

U.S. Pat. No. 6,658,059 to Iu et al;

U.S. Pat. No. 6,370,196 to Griessl et al;

US Published Patent Application 2005/0100207 of Konolige;

PCT Published Patent Application WO 98/03021 of SRI International;

PCT Published Patent Application WO 01/96982 of Dynapel Systems Inc.;

PCT Published Patent Application WO 05/006762 of Queen Mary & WestfieldCollege;

EP Patent Application No. EP 0 557 007 of Sony Corporation; and

EP Patent EP 0 966 727 to GMD—Forschungszentrum InformationstechnikGmbh.

The disclosures of all references mentioned above and throughout thepresent specification, as well as the disclosures of all referencesmentioned in those references, are hereby incorporated herein byreference.

SUMMARY OF THE INVENTION

The present invention seeks to provide an improved system and method fordetermining, in real time, a motion vector of an object imaged in astream of video image frames.

The present invention, in preferred embodiments thereof, enablesapplications to determine in a computationally efficient manner themotion vector of an object. In order to perform the computation in realtime, on a relatively weak CPU typical of set top boxes, use is made ofa mixture of optical principles, pattern matching, and stochasticaspects of object motion.

The determination of the motion vector is simplified by combining, in anoriginal way, different computational methods, converging to anestimation of the motion vector performed in real time, on low costhardware such as a set top box with a standard processor.

A preferred embodiment of the present invention has been developed forrunning on a set top box entitled “PILOTIME”. The “PILOTIME” set top boxis commercially available from THOMSON, 46 quai Alphonse Le Gallo, 92100Boulogne-Billancourt, France. The implementation on the PILOTIME set topbox demonstrates a real time implementation of optical flow concepts ona relatively common, relatively inexpensive device.

In accordance with a preferred embodiment of the present invention, themotion vector is determined as follows:

First, a zone within a first image frame is selected, the zonepreferably including a moving object. The zone is sub-divided intoblocks, termed herein frame elements. The frame elements include aplurality of pixels having associated pixel values. Within each frameelement, the pixel values are preferably normalized relative to eachframe element, to avoid luminance variation effects in the first imageframe.

It is noted that the motion vector is determined separately for eachframe element of the first image. The motion vector determined for frameelements depicting the moving object will preferably be substantiallyequal to the motion vector of the object, while frame elements notdepicting the moving object will preferably possess a motion vector of abackground.

Second, one or more reference pixels are typically selected in eachnormalized frame element.

Third, an optical flow is computed for each of the reference pixels,producing optic flow lines and an error range about the optic flowlines. The optical flow lines and the error range define a region.

Fourth, each normalized frame element of the first image frame has apattern matching step performed, the pattern matching being between thenormalized frame element of the first image frame and normalized testframe elements of a second, typically immediately following, imageframe. Displacements of the normalized test frame elements, relative tothe normalized frame element of the first image frame, are preferablyconfined to be within the region mentioned above. The results of thepattern matching are pattern matching scores and displacement vectors,which are associated with corresponding displacements of the test frameelement.

The lowest pattern matching score, which corresponds to the displacementwith a best fit, is selected, and the corresponding displacement vectoris used to calculate the motion vector of the object, thus determiningthe motion vector.

The present invention, in preferred embodiments thereof, generally usesapproximations to quadratic equations in order to simplify computations.

A typical use of the present invention, in preferred embodimentsthereof, is to evaluate, in real time, the motion vector of the objectin a scene produced by a webcam connected to a set top box, for example,but not limited to, enhancing interactive game applications, such asPong-Classic and Arkanoid-like games.

Preferred embodiments of the present invention enable high-end featuresto enrich user-STB interaction for sports, edutainment and fitnessgaming applications using low-cost hardware.

The term “image” in all its grammatical forms is used throughout thepresent specification and claims interchangeably with the terms “imageframe” and “digital image” and their corresponding grammatical forms.

There is thus provided in accordance with a preferred embodiment of thepresent invention a computer system for real-time determination of amotion vector of a frame element within an image frame in a stream ofvideo image frames, the stream including at least a first image frameand a second image frame, the system including a plurality ofoperatively connected elements including an image stream receiver toreceive the stream of image frames, an image decomposer to decomposeeach of the image frames into a plurality of frame elements, each of theframe elements including a plurality of pixels and a plurality ofassociated pixel values, an image normalizer to normalize the pixelvalues of the frame elements of each of the image frames therebyyielding a plurality of normalized frame elements having a plurality ofassociated normalized pixel values, an image processor to apply an imageprocessing filter to one of the normalized frame elements of the firstimage frame, yielding a filtered frame element having a plurality ofassociated filtered pixel values, a pixel selector to select a firstreference pixel from the filtered frame element, such that the firstreference pixel has the highest value of the filtered pixel values, anoptic flow module to determine a first optic flow based on an optic flowequation applied at a location of the first reference pixel, a patternmatching module to perform pattern matching between the normalized pixelvalues of the one normalized frame element of the first image frame, andthe normalized frame elements of the second image frame, being testframe elements, yielding a plurality of pattern matching scores, each ofthe test frame elements having one of the pattern matching scores,wherein the pattern matching is performed such that for each one of thetest frame elements, a displacement vector between a location of the onenormalized frame element and a location of the one test frame element isrestricted by a region which is at least partially based on the firstoptic flow, and a motion vector determiner to determine a motion vectorat least partially based on the displacement vector associated with oneof the test frame elements having a lowest one of the pattern matchingscores.

Further in accordance with a preferred embodiment of the presentinvention the stream of video image frames is a stream of color imageframes, and for each one of the pixels an associated one of the pixelvalues is a value of a luminosity component associated with the onepixel.

Still further in accordance with a preferred embodiment of the presentinvention the image processing filter is a contrast enhancing filter.

Additionally in accordance with a preferred embodiment of the presentinvention the contrast enhancing filter is a Sobel filter.

Moreover in accordance with a preferred embodiment of the presentinvention the pattern matching module computes the pattern matchingscore based on a sum of absolute values of differences between thenormalized pixel values of the one normalized frame element andcorresponding normalized pixel values of the one test frame element.

Further in accordance with a preferred embodiment of the presentinvention the pattern matching module defines the region as beingbounded by a first line and a second line, such that the first lineintersects the second line at an intersection, a third line bisects anangle between the first line and the second line, the third line is in adirection based at least partly on a direction of the first optic flow,and the third line and the first line subtend an angle epsilon.

Still further in accordance with a preferred embodiment of the presentinvention the pattern matching module determines the angle epsilonindependently of the first optic flow.

Additionally in accordance with a preferred embodiment of the presentinvention the pixel selector further selects a second reference pixelfrom the filtered frame element, the optic flow module furtherdetermines a second optic flow based on the optic flow equation appliedat a location of the second reference pixel, and the pattern matchingmodule is operative to perform a consistency check including checkingthat the angular difference between the second optic flow and the thirdline is less than, or equal to, the angle epsilon.

Moreover in accordance with a preferred embodiment of the presentinvention the pixel selector further selects a second reference pixelfrom the filtered frame element, such that the second reference pixelhas a second highest value of the filtered pixel values, and wherein theoptic flow module further determines a second optic flow based on theoptic flow equation applied at a location of the second reference pixel.

Further in accordance with a preferred embodiment of the presentinvention the pattern matching module determines the angle epsilon basedon an angular difference between the first optic flow and the secondoptic flow.

Still further in accordance with a preferred embodiment of the presentinvention the pixel selector further selects a third reference pixelfrom the filtered frame element, such that the third reference pixel hasa third highest value of the filtered pixel values, and wherein theoptic flow module further determines a third optic flow based on theoptic flow equation applied at a location of the third reference pixel.

Additionally in accordance with a preferred embodiment of the presentinvention the pattern matching module determines the angle epsilon atleast partly based on a value of an angle epsilon used in determiningthe motion vector of a previous normalized frame element of the firstimage frame.

Moreover in accordance with a preferred embodiment of the presentinvention the pattern matching module is operative to perform aconsistency check including checking that the angular difference betweenthe third optic flow and the first optic flow is less than, or equal to,the angle epsilon.

Further in accordance with a preferred embodiment of the presentinvention the pattern matching module restricts the region so that adistance between the intersection and any location within the region isless than a maximum value.

Still further in accordance with a preferred embodiment of the presentinvention the maximum value is one quarter of a smallest dimension of animage frame.

Additionally in accordance with a preferred embodiment of the presentinvention includes determining the maximum value by converting from amaximum physical speed to a corresponding displacement, the convertingbeing based at least partially on a frame rate of the stream of videoimage frames and at least one optical property.

Moreover in accordance with a preferred embodiment of the presentinvention the pattern matching module determines the maximum value basedon a maximum value of a previous frame element of the first image frameand on a motion vector of the previous frame element of the first imageframe.

Further in accordance with a preferred embodiment of the presentinvention the pattern matching module computes the distance as a sum ofan absolute value of a horizontal displacement and an absolute value ofa vertical displacement.

Still further in accordance with a preferred embodiment of the presentinvention if the lowest one of the pattern matching scores is greaterthan a first target score, the motion vector determiner determines themotion vector to be of zero magnitude.

Additionally in accordance with a preferred embodiment of the presentinvention the pattern matching module ceases pattern matching as soon asa pattern matching score is yielded which is less than a second targetscore.

Moreover in accordance with a preferred embodiment of the presentinvention the pattern matching module determines the second target scoreas 5% of an average of the pixel values of the one normalized frameelement of the first image frame.

Further in accordance with a preferred embodiment of the presentinvention the displacement vector having a beginning and an end, thebeginning of the display vector is at the intersection, and the patternmatching module performs a first pattern matching with a first one ofthe test frame elements with the end of the displacement vector at theintersection, and performs additional pattern matching with additionaltest frame elements by scanning the end of the displacement vectorthrough the region.

There is also provided in accordance with another preferred embodimentof the present invention a method for real-time determination, by aprocessor, of a motion vector of a frame element within an image framein a stream of video image frames, the stream including at least a firstimage frame and a second image frame, the method including receiving thestream of image frames, decomposing each of the image frames into aplurality of frame elements, each of the frame elements including aplurality of pixels and a plurality of associated pixel values,normalizing the pixel values of the frame elements of each of the imageframes thereby yielding a plurality of normalized frame elements havinga plurality of associated normalized pixel values, applying an imageprocessing filter to one of the normalized frame elements of the firstimage frame, yielding a filtered frame element having a plurality ofassociated filtered pixel values, selecting a first reference pixel fromthe filtered frame element, such that the first reference pixel has thehighest value of the filtered pixel values, determining a first opticflow based on an optic flow equation applied at a location of the firstreference pixel, performing pattern matching between the normalizedpixel values of the one normalized frame element of the first imageframe, and the normalized frame elements of the second image frame,being test frame elements, yielding a plurality of pattern matchingscores, each of the test frame elements having one of the patternmatching scores, wherein the pattern matching is performed such that foreach one of the test frame elements, a displacement vector between alocation of the one normalized frame element and a location of the onetest frame element is restricted by a region which is at least partiallybased on the first optic flow, and determining a motion vector at leastpartially based on the displacement vector associated with one of thetest frame elements having a lowest one of the pattern matching scores.

Further in accordance with a preferred embodiment of the presentinvention the pattern matching includes computing the pattern matchingscore based on a sum of absolute values of differences between thenormalized pixel values of the one normalized frame element andcorresponding normalized pixel values of the one test frame element.

Still further in accordance with a preferred embodiment of the presentinvention the region is bounded by a first line and a second line, thefirst line intersecting the second line at an intersection, a third linebisects an angle between the first line and the second line, the thirdline is in a direction based at least partly on a direction of the firstoptic flow, the third line and the first line subtend an angle epsilon.

Additionally in accordance with a preferred embodiment of the presentinvention includes selecting a second reference pixel from the filteredframe element, determining a second optic flow based on the optic flowequation applied at a location of the second reference pixel, andperforming a consistency check including checking that the angulardifference between the second optic flow and the third line is lessthan, or equal to, the angle epsilon.

Moreover in accordance with a preferred embodiment of the presentinvention includes selecting a second reference pixel from the filteredframe element, such that the second reference pixel has a second highestvalue of the filtered pixel values, determining a second optic flowbased on the optic flow equation applied at a location of the secondreference pixel, selecting a third reference pixel from the filteredframe element, such that the third reference pixel has a third highestvalue of the filtered pixel values, determining a third optic flow basedon the optic flow equation applied at a location of the third referencepixel, and performing a consistency check, including checking that theangular difference of the third optic flow from the first optic flow isless than, or equal to, the angle epsilon.

Further in accordance with a preferred embodiment of the presentinvention includes restricting the region so that a distance between theintersection and any location within the region is less than a maximumvalue.

Still further in accordance with a preferred embodiment of the presentinvention includes computing the distance as a sum of an absolute valueof a horizontal displacement and an absolute value of a verticaldisplacement.

Additionally in accordance with a preferred embodiment of the presentinvention includes determining the motion vector to be of zero magnitudeif the lowest one of the pattern matching scores is greater than a firsttarget score.

Moreover in accordance with a preferred embodiment of the presentinvention pattern matching ceases as soon as a pattern matching score isyielded which is less than a second target score.

Further in accordance with a preferred embodiment of the presentinvention the displacement vector having a beginning and an end, thebeginning of the display vector being at the intersection, andperforming a first pattern matching with a first one of the test frameelements, with the end of the displacement vector at the intersection,and additional pattern matching with additional test frame elements, byscanning the end of the displacement vector through the region.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood and appreciated more fully fromthe following detailed description, taken in conjunction with thedrawings in which:

FIG. 1 is a simplified pictorial illustration of a system constructedand operative in accordance with a preferred embodiment of the presentinvention;

FIG. 2A is a simplified pictorial illustration of an image depicting amoving object;

FIG. 2B is a simplified pictorial illustration of an image depicting themoving object of FIG. 2A and a motion vector of the moving objectdetermined by the system of FIG. 1;

FIG. 3 is a simplified flowchart illustration of a preferred method ofoperation of the system of FIG. 1;

FIG. 4 is a simplified pictorial illustration of a plurality ofreference pixels selected by the system of FIG. 1;

FIG. 5 is a simplified pictorial illustration of a plurality of opticflow lines produced by the system of FIG. 1;

FIG. 6 is a simplified pictorial illustration of a normalized frameelement of a first image frame, a displaced normalized test frameelement of a second image frame, and a scanning pattern fordisplacements used by the system of FIG. 1;

FIGS. 7A and 7B are simplified pictorial illustrations of lines used inrestricting a scanning pattern used by the system of FIG. 1; and

FIGS. 8A-8C are simplified pictorial illustrations of pattern matching,at different displacements, performed by the system of FIG. 1.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

Reference is now made to FIG. 1, which is a simplified pictorialillustration of a system 10 constructed and operative in accordance witha preferred embodiment of the present invention.

FIG. 1 depicts the system 10 in a typical example setting. The system 10is preferably implemented within a set top box 15. The set top box 15 istypically operatively connected to a video display 20, for example, butnot limited to, a television display. The set top box 15 is preferablyoperatively connected to a webcam 25. However, it will be appreciated bythose ordinarily skilled in the art that the set top box 15 can beconnected to any suitable digital video source, for example, but notlimited to, a broadcast, a point to point transmission of a video imageframe stream, and a playback of a recording. In the example of FIG. 1,the webcam 25 is positioned to provide a stream of video image frames ofa person 110 juggling a moving object 120.

The system 10 preferably includes the following functional modules:

an image stream receiver 30 preferably operative to receive a stream ofvideo image frames from a digital video source;

an image decomposer 35 typically operative to decompose images receivedfrom the image stream receiver 30, into sub-images, also termed frameelements, of N pixels by N pixels;

a normalizer 40 generally operative to normalize the frame elementsproduced by the image decomposer 35, as described in more detail belowwith reference to FIG. 3;

an image filter 45 preferably operative to apply an image processingfilter to the normalized frame elements produced by the normalizer 40,as described in more detail below with reference to FIG. 3;

a pixel selector 55 typically operative to select pixels which arereference pixels, or “locations of interest”, based on filtered frameelements produced by the image filter 45, as described in more detailbelow with reference to FIG. 3 and FIG. 4;

an optic flow module 60 generally operative to compute optic flows forat least some of the reference pixels, selected by the pixel selector55, as described in more detail below with reference to FIG. 3 and FIG.5;

a pattern matching module 62 preferably operative to perform patternmatching between a normalized frame element of a first image frame andnormalized test frame elements of a second image frame, producingpattern matching scores, as described below in more detail withreference to FIGS. 3, 6, 7A-7B, and 8A-8C; and

a motion vector determiner 65 typically operative to determine a motionvector based on pattern matching scores from the pattern matching module62.

Reference is now made to FIGS. 2A and 2B. FIG. 2A is a simplifiedpictorial illustration of an image frame 200 depicting the moving object120. FIG. 2B is a simplified pictorial illustration of an image 230depicting the moving object 120 of FIG. 2A and a motion vector 240 ofthe moving object 120 determined by the system 10 of FIG. 1.

FIG. 2A depicts the first image frame 200 of a stream of video imageframes, in which the person 110 is juggling the object 120.

FIG. 2B depicts the second image frame 230, from a later location in thestream of video image frames, preferably the very next image. Theposition of the object 120 in FIG. 2B has moved relative to the positionof the object 120 in FIG. 2A.

The motion vector 240 indicates the motion of the object 120 bydepicting the displacement of the object 120 from the location of object120 in image 200 to the location of object 120 in image 230.

In order to determine the motion vector 240 of the object 120 imaged inthe stream of video image frames, the following steps are preferablyperformed.

A stream of video image frames is provided to the system 10 of FIG. 1.

Reference is now made to FIG. 3, which is a simplified flowchartillustration of a preferred method of operation of the system 10 ofFIG. 1. Reference is also made to FIG. 1.

The stream of video image frames is preferably received by the imagestream receiver 30, including at least the first image frame 200 (FIG.2A) and the second image frame 230 (FIG. 2B). The first image frame 200and the second image frame 230 which preferably immediately follows thefirst image frame 200, are selected (step 300), by the image streamreceiver 30.

The decomposer 35 preferably decomposes the first image frame 200 andthe second image frame 230 into frame elements sized N pixels by Npixels (step 305).

The decomposer 35 preferably decomposes only zones of interest withinthe first image frame 200 and within the second image frame 230, inorder not to perform unnecessary processing on an entire image in caseswhere a partial image would suffice. The zones of interest are typicallyapplication specific. By way of a non-limiting example, for aPong-Classic game, the system 10 typically only searches for movementnear two opposite edges of the first image frame 200 and the secondimage frame 230, thereby necessitating decomposing the first image frame200 and the second image frame 230 near two opposite edges.

Each frame element of the images is defined by an N by N square matrix.For example, the p-th frame element is defined as:

$\begin{matrix}{{Element}_{p} = \begin{pmatrix}I_{1,1} & I_{1,2} & \ldots & I_{1,n} \\I_{2,1} & \ddots & \; & \vdots \\\vdots & \; & \ddots & \vdots \\I_{n,1} & \ldots & \ldots & I_{n,n}\end{pmatrix}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

wherein Ix,y is a value of a single pixel of an image, at frame elementcoordinates (x, y). The value of a pixel is described below.

Each frame element of the first image frame 200 and each frame elementof the second image frame 230 is now typically processed independentlyof other frame elements (step 310). Normalization is preferablyperformed to avoid luminance variation effects in the image, forexample, but not limited to, shadow, lighting, and/or noise. Anormalized frame element is produced by dividing each pixel value by anormalization factor specific to the frame element being processed (step320). The normalization factor ∥Norm_(p)∥ of the p-th frame element ispreferably defined by Equation 2 below:

$\begin{matrix}{{{Norm}_{p}} = {\frac{1}{n^{2}}{\sum\limits_{x = 1}^{n}{\sum\limits_{y = 1}^{n}I_{x,y}}}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

For gray scale images, the pixel value of a single pixel is typically asingle value of luminosity, within a range from black to white, as iswell known in the art.

For color images, each pixel typically comprises a set of three values.The three values typically define the pixel color and additional pixelproperties, using various color spaces, such as, for example, andwithout limiting the generality of the foregoing, the red, green, blue(RGB) color space; and the hue, luminance, saturation color space. Forcolor images, the three values of a color pixel are preferablytransformed to a color space comprising luminance, if needed. Then theluminance value is used as the pixel value.

A motion vector of an object is preferably determined separately foreach frame element in the image 200 (step 325). In cases where an objectis larger than a frame element, the object is typically associated witha plurality of motion vectors, each of the motion vectors preferablybeing uniquely associated with a frame element. In case where the objectis smaller than a frame element, only one frame element typically yieldsa motion vector corresponding to the motion of the object, while otherframe elements typically do not yield motion vectors of any significantmotion for the object, or yield motion vectors corresponding to themovement of other objects.

The size of a frame element is typically application specific, generallydepending on one or more of the following: the resolution of the digitalvideo source, the magnitude of the object and the size of the imageframe. By way of a non-limiting example, for a VGA frame (640 pixels by480 pixels) the size of each of the frame elements is 8 pixels by 8pixels, and for a High Definition TV (HDTV) frame, the size of each ofthe frame elements is 16 pixels by 16 pixels.

After normalization, a contrast enhancing filter, preferably a Sobelfilter, is typically applied to each of the normalized frame elements ofthe first image frame 200 (step 330), producing filtered normalizedframe elements of the first image frame 200. It will be appreciated bythose ordinarily skilled in the art that any suitable contrast enhancingfilter can be used, for example, but not limited to, a high-pass filter,and a deviation filter.

The p-th filtered normalized frame element Sp of the normalized frameelement p is determined using the following equation:

S _(P)=NormalizedElement_(p) •F _(sobel)  (Equation 3),

where the operands NormalizedElement_(p) and F_(sobel) are the p-thnormalized frame element of the image and the Sobel filter,respectively, and the • operator denotes the operation of applying afilter to a frame element, as is well known in the art.

By way of a non-limiting example, the Sobel filter used in a preferredembodiment of the present invention is:

$\begin{matrix}{F_{sobel} = \begin{pmatrix}{{- 1}/\sqrt{2}} & {- 1} & {{- 1}/\sqrt{2}} \\{- 1} & {4\left( {1 + {1/\sqrt{2}}} \right)} & {- 1} \\{{- 1}/\sqrt{2}} & {- 1} & {{- 1}/\sqrt{2}}\end{pmatrix}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

Further expansion of Equation 3 and Equation 4 provides:

S(x,y)4(1+1/√{square root over(2)})I′(x,y)−I′(x−1,y)−I′(x+1,y)−I′(x,y+1)−I′(x,y−1)−1/√{square rootover (2)}(I′(x+1,y+1)+I′(x−1,y−1)+I′(x−1,y+1)+I′(x+1,y−1))  (Equation 5)

wherein the I′(x,y) are the normalized values of pixels at coordinates(x, y) in the normalized frame elements of the first image frame 200.

Within each one of the filtered normalized frame elements of the firstimage frame 200, locations of local maxima of filtered pixel values arepreferably selected (step 335), producing locations of selected pixels,referred to as “reference pixels”. The step 335 of FIG. 3 is nowdescribed in more detail with reference to FIG. 4.

Reference is now made to FIG. 4 which is a simplified pictorialillustration of a plurality of reference pixels 420 selected by thesystem 10 of FIG. 1.

FIG. 4 depicts the first image frame 200 of FIG. 2A, and within thefirst image frame 200, a frame element 410. The frame element 410 isdepicted as having a size of 8 pixels by 8 pixels. Within the frameelement 410 several reference pixels 420 are depicted. The referencepixels 420 are located at locations of pixels which comprise localmaxima of the filtered normalized pixel values. The reference pixels 420are typically located at locations of high contrast within the frameelement 410.

The pixel selector 55 (FIG. 1) preferably selects three of the referencepixels 420, having the three highest pixel values of the filterednormalized frame element. However, in accordance with alternativepreferred embodiments of the present invention, one or more referencepixels 420 can be used, as will be described below with reference toFIGS. 7A and 7B.

Reference is now made to FIG. 5, which is a simplified pictorialillustration of a plurality of optic flow lines produced by the system10 of FIG. 1. Reference is also made to FIG. 3. FIG. 5 is an enlargedview of the region marked View V of FIG. 4.

FIG. 5 depicts the frame element 410, which includes a first referencepixel 510, a second reference pixel 520, a third reference pixel 530,and an additional reference pixel 540.

Three of the reference pixels, namely the reference pixels 510, 520, 530have the three highest filtered normalized pixel values in the frameelement 410. The first reference pixel 510 has the highest filterednormalized pixel value. The second reference pixel 520 has the secondhighest filtered normalized pixel value. The third reference pixel 530has the third highest filtered normalized pixel value. The additionalreference pixel 540 also has a filtered normalized pixel value which isa local maximum, as do the reference pixels 510, 520 and 530. Theadditional reference pixel 540 exemplifies that there can be more thanthree reference pixels in the filtered normalized frame element 410.

The optic flow module 60 preferably produces optic flows 515, 525, 535,for the selected reference pixels 510, 525, 530, by applying the opticflow equation (Equation 6 below) at the location of each of thereference pixels 510, 520, 530 respectively (step 340 of FIG. 3). A moredetailed description of how the optic flow module 60 produces the opticflows 515, 525, 535 is provided below with reference to Equation 6. Theoptic flow 515, the optic flow 525 and the optic flow 535 subtend anangle α₁ 517, an angle α₂ 527 and an angle α₃ 537, respectively, with ahorizontal reference direction.

The optic flow equation preferably used by the optic flow module 60(FIG. 1) is given below:

$\begin{matrix}{{\frac{{I^{\prime}\left( {{xi},{yi}} \right)}}{t} = {\left. 0\Leftrightarrow {\frac{\partial{I^{\prime}\left( {{xi},{yi}} \right)}}{\partial t} + {\frac{\partial{I^{\prime}\left( {{xi},{yi}} \right)}}{\partial x} \cdot \frac{\partial x}{\partial t}} + {\frac{\partial{I^{\prime}\left( {{xi},{yi}} \right)}}{\partial y} \cdot \frac{\partial y}{\partial t}}} \right. = 0}},} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

wherein I′(xi,yi) is a normalized value of a pixel of an image atcoordinates (x, y) of location i in the image.

It is to be appreciated that the formula above is most accurate when ∂t,∂x, and ∂y are small. In the case of a stream of video image frames, forexample: ∂t is generally considered to be small when ∂t is substantiallyequal to a time interval between consecutive image frames; and ∂x and ∂yare generally considered to be small when ∂x and ∂y are substantiallyequal to a size of one pixel.

The terms ∂x/∂t and ∂y/∂t of the equation 6, represent an x componentV_(x) and a y component V_(y) of a motion vector V, respectively.

The equation 6 is preferably simplified by defining the time differencebetween the first image frame 200 and the second image frame 230 as oneunit of time, such that ∂t=1.

Substituting V_(x) and V_(y) as well as ∂t=1 into the equation 6 givesan equation of a straight line, as follows:

V _(y)(i)=a(i)V _(x)(i)+b(i)  (Equation 7)

wherein a(i) and b(i) are slope and intercept coefficients,respectively, of a straight line, produced by the optic flow equation 7applied to the image at a location i.

It is to be appreciated that Equation 7 is an equation with twounknowns, V_(x) and V_(y).

Equation 7 is now solved separately for a(i) and for b(i):

$\begin{matrix}{{b(i)} = \frac{{I_{1}^{\prime}\left( {{xi},{yi}} \right)} - {I_{2}^{\prime}\left( {{xi},{yi}} \right)}}{{I_{2}^{\prime}\left( {{xi},{yi}} \right)} - {I_{2}^{\prime}\left( {{xi},{{yi} - 1}} \right)}}} & \left( {{Equation}\mspace{14mu} 8} \right) \\{{a(i)} = \frac{{I_{2}^{\prime}\left( {{xi},{yi}} \right)} - {I_{2}^{\prime}\left( {{{xi} + 1},{yi}} \right)}}{{I_{2}^{\prime}\left( {{xi},{{yi} + 1}} \right)} - {I_{2}^{\prime}\left( {{xi},{yi}} \right)}}} & \left( {{Equation}\mspace{14mu} 9} \right)\end{matrix}$

wherein I′₁(xi,yi) is preferably a normalized value of a pixel of thefirst image at coordinates (xi,yi); I′₂(xi,yi) is preferably anormalized value of a pixel of the second image frame at coordinates(xi,yi); and I′₂(xi,yi−1) is preferably a normalized value of a pixel ofthe second image frame at coordinates (xi,yi−1), that is, a locationhaving a y coordinate one less than the y coordinate of I′₂(xi,yi). Itwill be appreciated by those ordinarily skilled in the art that theother terms of equations 8 and 9 are defined in substantially the sameway with suitable modifications with respect to the coordinate values.

By way of a non-limiting example, I′₁(xi,yi) is preferably a normalizedvalue of the first reference pixel 510 of the first image frame 200,I′₂(xi,yi) is preferably a normalized value of a pixel of the secondimage frame 230, which is located at the location of the first referencepixel 510, I′₂(xi,yi−1) is preferably a normalized value of a pixel ofthe second image frame 230, which is located below of the firstreference pixel 510, and so forth. It will be appreciated by thoseordinarily skilled in the art that the other terms of equations 8 and 9are exemplified in substantially the same way with suitablemodifications with respect to the coordinate values.

Therefore, optic flow lines 515, 525, 535, are each preferably producedby applying Equation 8 and Equation 9 to the normalized frame element410 of the first image frame 200 at the reference pixels 510, 520, 530,respectively, thereby determining a(i) and b(i) for each of thereference pixels 510, 520, 530. The calculated values a(i) and b(i) arepreferably substituted into Equation 7 for each of the reference pixels510, 520, 530, in order to determine the optic flow lines 515, 525, 535,respectively.

It should be noted that equations 7-9, used to determine the slope a(i)and intercept b(i) of the optic flows 515, 525, 535, define a straightline for each of the reference pixels 510, 520, 530. However, themagnitude of the optic flows 510, 520, 530 is not typically known.

Therefore, in general, the optic flow lines are only typically limitedby the edges of the image frames.

Reference is now made to FIG. 6, which is a simplified pictorialillustration of the normalized frame element 410 of the first imageframe 200, a displaced normalized test frame element 650 of the secondimage frame 230 (FIG. 2B), and a scanning pattern 629 for displacementsused by the system 10 of FIG. 1. Reference is also made to FIGS. 1 and3.

By way of introduction, the pattern matching module 62 preferablyperforms pattern matching between the normalized frame element 410 ofthe first image frame 200 and normalized test frame elements 650 of thesecond image frame 230. Therefore, the pattern matching module 62preferably instructs:

the decomposer 35 to produce test frame elements of the second imageframe 230; and

the normalizer 40 to normalize the test frame elements, producingnormalized test frame elements 650 of the second image frame 230 atvarious displacements relative to the frame element 410 of the firstimage frame 200, as needed during pattern matching.

The displacements are preferably incremented by one pixel at a time. Onesuch displacement, by way of a non-limiting example, is depicted by adisplacement vector 660, which extends from the location of the frameelement 410 of the first image frame 200, to the location of thenormalized test frame element 650 of the second image frame 230.

Efficiency is preferably gained by performing the pattern matching onlyin a region 625 where a good match is expected, thereby reducingunnecessary computation. Determining the region 625 is described in moredetail below with reference to FIGS. 6, 7A and 7B.

FIG. 6 depicts the normalized frame element 410, and two lines, a firstline 610 and a second line 620, intersecting at an intersection 630. Thedisplacements mentioned above are typically measured as displacementsfrom the intersection 630. The method of determining the first line 610and the second line 620 is described in more detail below with referenceto FIGS. 7A and 7B.

It is to be appreciated that the location of the frame element 410 andthe location of the test frame element 650 are typically defined withrespect to a predetermined pixel within the frame elements. The fixedpixel is preferably a specific corner, the corner being the same cornerin both frame elements. The intersection 630 is preferably located atthe location of the normalized frame element 410.

The pattern matching module 62 performs pattern matching between thetest frame element 650 and the frame element 410, as described in moredetail below with reference to Equation 15, producing a pattern matchingscore (step 345 of FIG. 3).

FIG. 6 additionally depicts vectors, which are directed line segments,drawn between the first line 610 and the second line 620. The vectorsdescribe the scanning pattern 629 in which the test frame elements 650are typically displaced from the intersection 630. The pattern matchingmodule 62 (FIG. 1) begins scanning at a displacement of zero,corresponding to the intersection 630. The pattern matching moduletypically continues scanning by changing the displacement by one pixelincrements along vector 631, then along vector 632, then along vector633, then along vector 634, and so forth, until a stop condition hasbeen reached. The stop condition generally comprises one of: a goodenough match has been achieved; and the entire region 625 has beenscanned. The stop condition is described in more detail below withreference to FIGS. 8A-8C.

The pattern matching module 62 typically instructs the image decomposer35 to produce the test frame element 650 of the second image frame 230at each displacement in the scanning pattern. The pattern matchingmodule 62 also typically instructs the normalizer 40 to normalize thetest frame elements 650, producing normalized test frame elements 650.The pattern matching module 62 then performs pattern matching, asdescribed in greater detail with reference to FIGS. 8A-8C below. Thepattern matching module 62 produces a pattern matching score andassociates the pattern matching score with the displacement vector 660corresponding to the displacement of the test frame element 650.

In accordance with a most preferred embodiment of the present invention,in addition to scanning in the region 625 confined between two lines, alimit is also imposed on what distance from the intersection 630 theregion 625 may extend. The limit (or maximum value of the distance) is,at least initially, typically, a distance of one quarter of an imageframe. Since image frames are typically rectangular, the limit isgenerally a distance of one quarter of a smaller dimension of an imageframe.

In accordance with the most preferred embodiment of the presentinvention, when a motion vector is determined for a prior video imageframe, the pattern matching module 62 typically records a value of thedisplacement vector 660 associated with the motion vector. When asubsequent determination, using the first image frame 200 and the secondimage frame 230, both taken from later in the stream of video imageframes, is performed, the limit is generally twice the previouslyrecorded value of the displacement vector.

It is to be appreciated that limiting the region 625 greatly savescomputation.

In accordance with another preferred embodiment of the presentinvention, scanning is limited to a distance preferably based on: ascanning limit used for a previous frame element of the first imageframe 200; and on a displacement corresponding to the motion vectordetermined for the previous frame element. The scanning limit is thusadaptable, based on a previous scanning limit, and modified by an actualdistance to a previous “best-fit” test frame.

One non-limiting example of how the scanning is limited as describedabove is:

${L_{p} = {\frac{\left( {L_{p - 1} + D_{p - 1}} \right)}{2} \star 1.2}},$

where: L_(p) is a scanning limit for frame element p; L_(p−1) is ascanning limit from an immediately previous frame element p−1; andD_(p−1) is a displacement corresponding to the motion vector determinedfor frame element p−1. Typical scanning limits and displacements are:

p 1 2 3 4 5 6 7 L_(p) [pixels] 100 (an initial value) 63 41 31 28 32 34D_(p−1) [pixels] 5 5 10 15 25 25 25It is to be appreciated that for p=1, which corresponds to a first frameelement, an initial value is chosen as the scanning limit L₁. Thetypical scanning limits show that even while the displacement varies(from 5 pixels to 25 pixels), the scanning limit converges rapidly to avalue close to the displacement. Thus the scanning limit provides agreat saving in computation, while maintaining a region large enough todetermine the displacement.

D is preferably determined as described below.

Preferably, an optimization is used to calculate the distance D. Sincethe displacement of the test frame element 650 needs to be calculatedand compared to the scanning limit, optimizing the calculation of thedistance D is typically beneficial. The distance D between two pixels,p₁ and p₂, in the image frame is preferably calculated asD=|x₁−x₂|+y₁−y₂|, where x₁ is the x-coordinate of pixel p₁; x₂ is thex-coordinate of pixel p₂; y₁ is the y-coordinate of pixel p₁; and y₂ isthe y-coordinate of pixel p₂.

In accordance with an alternative preferred embodiment of the presentinvention the distance D is calculated as:

D=√{square root over ((x ₁ −x ₂)²+(y ₁ −y ₂)².)}{square root over ((x ₁−x ₂)²+(y ₁ −y ₂)².)}

The optimization of the preferred method of calculating the distance Dgreatly cuts down computing load, as operations of computing an absolutevalue and adding are less computation intensive than operations ofmultiplying and computing a square root.

It is to be appreciated that for every displacement vector there existsa corresponding motion vector. Since displacement divided by time equalsvelocity, dividing the displacement vector 660 by the time between thefirst image frame 200 and the second image frame 230, ∂t, produces amotion vector, whose units are pixels per unit of time. The motionvector is generally an expression of actual movement of the object in areal world, and is subject to physical constraints. The magnitude of themotion vector, which corresponds to velocity, is typically required tobe within a range of physically possible velocities.

Equation 10 below defines a typical magnitude constraint for motionvectors:

√{square root over (Vx ² +Vy ²)}≦V _(max)  (Equation 10)

wherein Vmax is a practical upper limit to the motion vector, specificto each application. Typically, Vmax corresponds to the maximumphysically possible velocity mentioned above. For example, the maximumphysically possible velocity on an application intended for use in atennis game, is typically the maximum velocity of a tennis ball in atennis game. Alternatively, Vmax may correspond to a velocity equal to adisplacement of one quarter frame in one inter-frame time period.

By way of a non-limiting example, typical motion vectors measured by asystem imaging at VGA resolution, at a rate of 20 frames per second, ofsomeone juggling objects at a distance of approximately 4 meters from awebcam, are measured to be approximately 10 pixels per 1/20 second.

It is well known in the art to convert from physical speed Vmax to amaximum value of a motion vector, based on optical properties such as atype of camera lens and distance of an object from the camera lens, andto convert from the maximum value of the motion vector to a displacementbased on a frame rate of the stream of video image frames. Therefore,the scanning limit (or maximum value of the distance for scanning) ispreferably determined by converting from a maximum physical speed to acorresponding displacement, wherein the converting is typically based ona frame rate of the stream of video image frames and optical propertiesof the webcam 25.

It is also well-known in the art, given a frame rate of a stream ofvideo image frames, to determine a displacement vector from acorresponding motion vector, and vice versa.

In accordance with a preferred embodiment of the present invention, thescoring process is generally optimized by invalidating displacementvectors which fail the constraint of Equation 10 above.

Reference is now made to FIGS. 7A and 7B, which are simplified pictorialillustrations of lines used in restricting a scanning pattern used bythe system 10 of FIG. 1. Reference is also made to FIG. 1 and to FIG. 5.

FIG. 7A depicts the first line 610 and the second line 620, and theintersection 630. As described with reference to FIG. 6, the first line610 and the second line 620 define the region 625 within which patternmatching is performed. FIG. 7A also depicts a third line 720 whichbisects the angle between the first line 610 and the second line 620,producing an angle ε 730 between the first line 610 and the third line720, and the angle ε 730 between the second line 620 and the third line720. FIG. 7A additionally depicts the first optic flow line 515, thesecond optic flow line 525, and the third optic flow line 535, drawnthrough the intersection 630, and corresponding angles α₁ 517, α₂ 527and α₃ 537, measured from a horizontal reference line 710.

Therefore, the region 625 is determined from the third line 720, theangle ε 730, and the location of the intersection 630 such that thefirst line 610 and the second line 620 pass through the intersection 630and each of the lines 610, 620 are displaced from the third line 720 bythe angle ε 730.

The pattern matching module 62 preferably: computes the third line 720to be the average of the three optic flow lines 515, 525, and 535;determines the angle ε 730 as a measure of variability of the optic flowlines 515, 525, 535 about the direction of the third line 720; anddetermines the location of the intersection 630 as corresponding to zerodisplacement from the location of the frame element 410.

Determining the measure of variability is now described below.

It is to be appreciated that the a(i) and the b(i) (calculated byEquations 9 and 8), being empirical results, comprise some variability,caused, among other causes, by digitizing errors and by artifacts withinthe first image frame 200 and the second image frame 230. Therefore, inaccordance with a most preferred embodiment of the present invention, aset-up step is used to perform a measure of variability. A user, imagedby the webcam 25 operatively connected to the system 10, is instructedto perform movements. The system 10 goes through the process ofdetermining a motion vector, during which the optic flow module 60produces optic flow lines, and the pattern matching module 62 determinesthe variability between the optic flow lines, and uses the variabilityto determine the angle ε.

Alternatively, if a measurement of variability is not performed withinthe set-up step, default values are used for the variability of both aand b. The default values for the variability of a and b are shownbelow, in Equations 11 and 12.

$\begin{matrix}{{{variability}\mspace{14mu} {of}\mspace{14mu} a} = {{\pm \frac{1}{n}}\sqrt{\sum\limits_{i = 1}^{n}\left( {a_{i} - \overset{\_}{a}} \right)^{2}}}} & \left( {{Equation}\mspace{14mu} 11} \right) \\{{{variability}\mspace{14mu} {of}\mspace{14mu} b} = {{\pm \frac{1}{n}}\sqrt{\sum\limits_{i = 1}^{n}\left( {b_{i} - \overset{\_}{b}} \right)^{2}}}} & \left( {{Equation}\mspace{14mu} 12} \right)\end{matrix}$

wherein a_(i) and b_(i) are the a and b components of an i-th optic flowline, and ā and b are the mean a and b components of n optic flow lines,and n is the number of optic flow lines included in the calculation.Variability is typically measured among the three optic flow lines 515,525, and 535.

In an alternative preferred embodiment of the present invention, anoptimization is preferably used to calculate the variability, as shownbelow, in Equation 13 and 14.

$\begin{matrix}{{{variability}\mspace{14mu} {of}\mspace{14mu} a} = {{\pm \frac{1}{n}}{\sum\limits_{i = 1}^{n}{{a_{i} - \overset{\_}{a}}}}}} & \left( {{Equation}\mspace{14mu} 13} \right) \\{{{variability}\mspace{14mu} {of}\mspace{14mu} b} = {{\pm \frac{1}{n}}{\sum\limits_{i = 1}^{n}{{b_{i} - \overset{\_}{b}}}}}} & \left( {{Equation}\mspace{14mu} 14} \right)\end{matrix}$

wherein the variables are defined as in Equations 11 and 12.

By way of example only, empirical results show that at a frame rate of20-25 image frames per second, someone juggling objects at a distance ofapproximately 4 meters from an imaging video camera, the variability ofb is typically ±10°. The variability is typically inversely proportionalto the frame rate.

Default values for variability are likely to change according tospecific applications. For example, and without limiting the generalityof the foregoing, in applications in which the motion vector is expectedto have only a vertical component, such as Pong-Classic, the third line720 is preferably defined as having a zero horizontal component and novariability is typically taken into account in the horizontal direction.

Reference is now made to FIG. 7B. In accordance with another preferredembodiment of the present invention, the pattern matching module 62defines the region 625, as follows:

the second optic flow line 525 defines one border of the region 625;

the first optic flow line 515 bisects the region 625 (FIG. 6);

the angle ε 730 is defined as the angle between the first optic flowline 515 and the second optic flow line 525; and

the second line 620 is defined to be at the angle ε 730 from the firstoptic flow line 515, on the side opposite to the second optic flow line525. The second line 620 thus defines a second border of the region 625.

Optionally, a consistency check is performed by the pattern matchingmodule 62 comparing an angle ε₁ between the third optic flow line 535and the first optic flow line 515 to the angle ε. If the angle ε₁ issmaller than or equal to the angle ε 730, then the pattern matchingmodule 62 generally determines that the optic flow lines are consistent.If the third optic flow line happens to be at an angle ε₂ which isgreater than the angle ε 730, as depicted by the angle ε₂ to a lowerline 740, then the pattern matching module 62 generally determines thatoptic flow lines are not consistent.

It will be appreciated by those ordinarily skilled in the art that theabove consistency check can be performed on the region 625 describedwith reference to FIG. 7A. By way of a non-limiting example, the patternmatching module compares an angle subtended by at least one of the opticflow lines 515, 525, and 535 with the third line 720, and if the anglesubtended is greater than the angle ε then the pattern matching module62 generally determines that optic flow lines are not consistent.

Reference is now made to FIGS. 8A-8C, which are simplified pictorialillustrations of pattern matching, at different displacements, performedby the system 10 of FIG. 1. Reference is also made to FIG. 1, FIG. 2,and FIG. 3.

FIG. 8A depicts the person 110, the object 120, and the frame element410 containing part of the object 120, in the first image frame 200.

FIG. 8B depicts the person 110 and the object 120 in the second imageframe 230. The object 120 of FIG. 8B is located at a different locationwithin the second image frame 230 than the location of the object 120within the first image frame 200 of FIG. 8A. FIG. 8B depicts thelocation of the frame element 410 of FIG. 8A and a test frame element810 located at a location displaced by a displacement vector 815 fromthe frame element 410.

It is to be appreciated that the pattern matching score, produced by thepattern matching module 62, is a measure of how well the test frameelement 810, displaced by the displacement vector 815, fits theunderlying image 230.

The pattern matching score is preferably calculated based on a sum ofsquares of differences between the normalized pixel values of the firstframe element 410 and corresponding normalized pixel values of the testframe element 810, typically using Equation 15:

$\begin{matrix}{{{Score}_{p}\left( {{Vx},{Vy}} \right)} = {\sum\limits_{x = 1}^{n}{\sum\limits_{y = 1}^{n}{\begin{pmatrix}{\left( {{I_{1\; p}^{\prime}\left( {x,y} \right)} - {{Norm}_{1\; p}}} \right) -} \\\begin{pmatrix}{{I_{2\; p}^{\prime}\left( {{x + {Vx}},{y + {Vy}}} \right)} -} \\{{N_{{orm}_{2\; p}}\left( {{x + {Vx}},{y + {Vy}}} \right)}}\end{pmatrix}\end{pmatrix}^{2}.}}}} & \left( {{Equation}\mspace{14mu} 15} \right)\end{matrix}$

In Equation 15 the following notation is used:

Vx and Vy are the x and y components of the displacement vector 815;

∥Norm_(2p)(x+Vx,y+Vy)∥ is a normalization factor (Equation 2) for frameelement p, being the test frame element 810 of the second image frame230, wherein the test frame element 810 is displaced by Vx in thex-direction and by Vy in the y-direction, relative to the first frameelement 410;

I′_(2p)(x+Vx,y+Vy) is a pixel value at coordinates x+Vx, y+Vy of thetest frame element 810 of the second image frame 230;

∥Norm_(1p)∥ is the normalization factor (Equation 2) for frame element pof the frame element 410 of the first image frame 200; and

I′_(1p)(x, y) is the pixel value at coordinates x, y of the frameelement 410 of the first image frame 200.

In accordance with another preferred embodiment of the present inventionthe pattern matching score is preferably calculated based on a sum ofabsolute values of differences between the normalized pixel values ofthe first frame element 410 and corresponding normalized pixel values ofthe test frame element 810, typically using Equation 16:

$\begin{matrix}{{{Score}_{p}\left( {{Vx},{Vy}} \right)} = {\sum\limits_{x = 1}^{n}{\sum\limits_{y = 1}^{n}{\begin{matrix}{\left( {{I_{1\; p}^{\prime}\left( {x,y} \right)} - {{Norm}_{1\; p}}} \right) -} \\\begin{pmatrix}{{I_{2\; p}^{\prime}\left( {{x + {Vx}},{y + {Vy}}} \right)} -} \\{{N_{{orm}_{2\; p}}\left( {{x + {Vx}},{y + {Vy}}} \right)}}\end{pmatrix}\end{matrix}}}}} & \left( {{Equation}\mspace{14mu} 16} \right)\end{matrix}$

The notation used for Equation 16 is the same as the notation used forEquation 15.

The pattern matching score is generally inversely proportional to adegree of similarity between the normalized frame element 410 of thefirst image frame 200 and the normalized test frame element 810 of thesecond image frame 230. The lower the pattern matching score is, themore similar the normalized frame element 410 in the first image frame200 is to the normalized test frame element 810 in the second imageframe 230. In FIG. 8B the object 120 is partially within the test frameelement 810. Additionally, in FIG. 8B, the object 120 is located in asimilar fashion with respect to the test frame element 810 as the object120 of FIG. 8A is located with respect to the frame element 410 of FIG.8A. Therefore, similarity is high between the normalized frame element410 and the normalized test frame element 810, and the pattern matchingmodule 62 preferably produces a low score.

FIG. 8C depicts the person 110 and the object 120 in the second imageframe 230. The object 120 of FIG. 8C is located at the same imagecoordinates as the object 120 of FIG. 8B. FIG. 8C depicts the locationof the frame element 410 of FIG. 8A, and a test frame element 820 at alocation displaced by a displacement vector 825 from a location of theframe element 410. In FIG. 8C the object 120 is within the test frameelement 810, but located quite differently from the object 120 in theframe element 410 of FIG. 8A. Similarity is low, and the patternmatching module 62 (FIG. 1) typically produces a high score.

In the example of FIGS. 8A-8C, it is to be appreciated that the scorewhich the displacement vector 825 of FIG. 8C receives is higher than thescore which displacement vector 815 of FIG. 8B receives, because thedisplaced test frame element 820 does not fit the image 230 as well asthe displaced test frame element 810.

Once pattern matching scores corresponding to displacements of testframe elements have been computed, the pattern matching module 62preferably selects a displacement vector 815 having a lowest patternmatching score (step 350 of FIG. 3). The motion vector determiner 65then determines the motion vector based on the displacement vector 815,by dividing the displacement vector 815 by the time between the firstimage frame 200 and the second image frame 230 (step 355).

The motion vector determiner 65 preferably determines the motion vectorto be of zero magnitude if the pattern matching module produces patternmatching scores which are all greater than a first target score. Thefirst target score is typically equal to 15% of an average of thenormalized pixel values of the first frame element 410 of the firstimage frame 200.

Additionally, the pattern matching module 62 preferably ceases scanningupon yielding a pattern matching score less than a second target score,and the motion vector determiner 65 preferably determines the motionvector based on the first displacement which yields the pattern matchingscore less than the second target score. The second target score istypically equal to 5% of an average of the normalized pixel values ofthe first frame element 410 of the first image frame 200.

The method described above is preferably used to determine a firstmotion vector using a first image frame 200 and a second image frame230, and used again to determine a second motion vector using a thirdimage and a fourth image. If the first motion vector and the secondmotion vector are substantially equal, the value of the motion vector ispreferably considered consistent. Otherwise, additional determinationsof motion vectors are preferably performed until two consecutivedeterminations of motion vectors, for different pairs of images, aresubstantially equal.

By way of a non-limiting example, the example application alreadydescribed above with reference to FIGS. 6 and 7A is again considered.The example application comprises the system 10 of FIG. 1 imaging aperson juggling objects at a distance of approximately 4 meters from awebcam. In the example application, the system 10 of FIG. 1 defines afirst motion vector substantially equal to a second motion vector when

${{2 \star \frac{{Len}\left( {V_{1} - V_{2}} \right)}{{Len}\left( {V_{1} + V_{2}} \right)}} < {30\%}},$

where V₁ is the first motion vector, V₂ is the second motion vector, andLen is a function measuring the length of a vector. The function Len(V)is defined as:

Len(V)=|V _(x) |+|V _(y)|.

It is to be appreciated that determining the second motion vector byusing a third image and a fourth image can be done by using the secondimage frame 230 of the first determination as the first image frame 200of the second determination, and using the third image as the secondimage frame 230 of the second determination.

The motion vector determiner 65 generally determines the motion vectorin units of, by way of a non-limiting example, pixels per second. It iswell known in the art that further conversions of the motion vector toother units, such as, by way of a non-limiting example, meters persecond, are easily performed based on optical and physical data of thedigital video source.

It is appreciated that software components of the present invention may,if desired, be implemented in ROM (read only memory) form. The softwarecomponents may, generally, be implemented in hardware, if desired, usingconventional techniques.

It is appreciated that various features of the invention which are, forclarity, described in the contexts of separate embodiments may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention which are, for brevity, described in thecontext of a single embodiment may also be provided separately or in anysuitable subcombination.

It will be appreciated by persons skilled in the art that the presentinvention is not limited by what has been particularly shown anddescribed hereinabove. Rather the scope of the invention is defined onlyby the claims which follow:

1. A computer system for real-time determination of a motion vector of aframe element within an image frame in a stream of video image frames,the stream including at least a first image frame and a second imageframe, the system comprising a plurality of operatively connectedelements including: an image stream receiver to receive the stream ofimage frames; an image decomposer to decompose each of the image framesinto a plurality of frame elements, each of the frame elements includinga plurality of pixels and a plurality of associated pixel values; animage normalizer to normalize the pixel values of the frame elements ofeach of the image frames thereby yielding a plurality of normalizedframe elements having a plurality of associated normalized pixel values;an image processor to apply an image processing filter to one of thenormalized frame elements of the first image frame, yielding a filteredframe element having a plurality of associated filtered pixel values; apixel selector to select a first reference pixel from the filtered frameelement, such that the first reference pixel has the highest value ofthe filtered pixel values; an optic flow module to determine a firstoptic flow based on an optic flow equation applied at a location of thefirst reference pixel; a pattern matching module to perform patternmatching between the normalized pixel values of: the one normalizedframe element of the first image frame; and the normalized frameelements of the second image frame, being test frame elements, yieldinga plurality of pattern matching scores, each of the test frame elementshaving one of the pattern matching scores, wherein the pattern matchingis performed such that for each one of the test frame elements, adisplacement vector between a location of the one normalized frameelement and a location of the one test frame element is restricted by aregion which is at least partially based on the first optic flow; and amotion vector determiner to determine a motion vector at least partiallybased on the displacement vector associated with one of the test frameelements having a lowest one of the pattern matching scores.
 2. Thesystem of claim 1 and wherein the stream of video image frames is astream of color image frames, and for each one of the pixels anassociated one of the pixel values is a value of a luminosity componentassociated with the one pixel.
 3. The system of claim 1 and wherein theimage processing filter is a contrast enhancing filter.
 4. The system ofclaim 3 and wherein the contrast enhancing filter is a Sobel filter. 5.The system of claim 1 and wherein the pattern matching module computesthe pattern matching score based on a sum of absolute values ofdifferences between the normalized pixel values of the one normalizedframe element and corresponding normalized pixel values of the one testframe element.
 6. The system of claim 1 and wherein the pattern matchingmodule defines the region as being bounded by a first line and a secondline, such that: the first line intersects the second line at anintersection; a third line bisects an angle between the first line andthe second line; the third line is in a direction based at least partlyon a direction of the first optic flow; and the third line and the firstline subtend an angle epsilon.
 7. The system of claim 6 and wherein thepattern matching module determines the angle epsilon independently ofthe first optic flow.
 8. The system of claim 6 and wherein: the pixelselector further selects a second reference pixel from the filteredframe element; the optic flow module further determines a second opticflow based on the optic flow equation applied at a location of thesecond reference pixel; and the pattern matching module is operative toperform a consistency check including checking that the angulardifference between the second optic flow and the third line is lessthan, or equal to, the angle epsilon.
 9. The system of claim 6 andwherein the pixel selector further selects a second reference pixel fromthe filtered frame element, such that the second reference pixel has asecond highest value of the filtered pixel values, and wherein the opticflow module further determines a second optic flow based on the opticflow equation applied at a location of the second reference pixel. 10.The system of claim 9 and wherein the pattern matching module determinesthe angle epsilon based on an angular difference between the first opticflow and the second optic flow.
 11. The system of claim 9 and whereinthe pixel selector further selects a third reference pixel from thefiltered frame element, such that the third reference pixel has a thirdhighest value of the filtered pixel values, and wherein the optic flowmodule further determines a third optic flow based on the optic flowequation applied at a location of the third reference pixel.
 12. Thesystem of claim 6 and wherein the pattern matching module determines theangle epsilon at least partly based on a value of an angle epsilon usedin determining the motion vector of a previous normalized frame elementof the first image frame.
 13. The system of claim 11 and wherein thepattern matching module is operative to perform a consistency checkincluding checking that the angular difference between the third opticflow and the first optic flow is less than, or equal to, the angleepsilon.
 14. The system of claim 6 and wherein the pattern matchingmodule restricts the region so that a distance between the intersectionand any location within the region is less than a maximum value.
 15. Thesystem of claim 14 and wherein the maximum value is one quarter of asmallest dimension of an image frame.
 16. The system of claim 14 andfurther comprising determining the maximum value by converting from amaximum physical speed to a corresponding displacement, the convertingbeing based at least partially on a frame rate of the stream of videoimage frames and at least one optical property.
 17. The system of claim14 and wherein the pattern matching module determines the maximum valuebased on a maximum value of a previous frame element of the first imageframe and on a motion vector of the previous frame element of the firstimage frame.
 18. The system of claim 14 and wherein the pattern matchingmodule computes the distance as a sum of an absolute value of ahorizontal displacement and an absolute value of a verticaldisplacement.
 19. The system of claim 1 and wherein if the lowest one ofthe pattern matching scores is greater than a first target score, themotion vector determiner determines the motion vector to be of zeromagnitude.
 20. The system of claim 1 and wherein the pattern matchingmodule ceases pattern matching as soon as a pattern matching score isyielded which is less than a second target score.
 21. The system ofclaim 20 and wherein the pattern matching module determines the secondtarget score as 5% of an average of the pixel values of the onenormalized frame element of the first image frame.
 22. The system ofclaim 1 and wherein the displacement vector having a beginning and anend, the beginning of the display vector is at the intersection, and thepattern matching module: performs a first pattern matching with a firstone of the test frame elements with the end of the displacement vectorat the intersection; and performs additional pattern matching withadditional test frame elements by scanning the end of the displacementvector through the region.
 23. A method for real-time determination, bya processor, of a motion vector of a frame element within an image framein a stream of video image frames, the stream including at least a firstimage frame and a second image frame, the method comprising: receivingthe stream of image frames; decomposing each of the image frames into aplurality of frame elements, each of the frame elements including aplurality of pixels and a plurality of associated pixel values;normalizing the pixel values of the frame elements of each of the imageframes thereby yielding a plurality of normalized frame elements havinga plurality of associated normalized pixel values; applying an imageprocessing filter to one of the normalized frame elements of the firstimage frame, yielding a filtered frame element having a plurality ofassociated filtered pixel values; selecting a first reference pixel fromthe filtered frame element, such that the first reference pixel has thehighest value of the filtered pixel values; determining a first opticflow based on an optic flow equation applied at a location of the firstreference pixel; performing pattern matching between the normalizedpixel values of: the one normalized frame element of the first imageframe; and the normalized frame elements of the second image frame,being test frame elements, yielding a plurality of pattern matchingscores, each of the test frame elements having one of the patternmatching scores, wherein the pattern matching is performed such that foreach one of the test frame elements, a displacement vector between alocation of the one normalized frame element and a location of the onetest frame element is restricted by a region which is at least partiallybased on the first optic flow; and determining a motion vector at leastpartially based on the displacement vector associated with one of thetest frame elements having a lowest one of the pattern matching scores.24-32. (canceled)
 33. A computer system for real-time determination of amotion vector of a frame element within an image frame in a stream ofvideo image frames, the stream including at least a first image frameand a second image frame, the system comprising a plurality ofoperatively connected elements including: means for receiving the streamof image frames; means for decomposing each of the image frames into aplurality of frame elements, each of the frame elements including aplurality of pixels and a plurality of associated pixel values; meansfor normalizing the pixel values of the frame elements of each of theimage frames thereby yielding a plurality of normalized frame elementshaving a plurality of associated normalized pixel values; means forapplying an image processing filter to one of the normalized frameelements of the first image frame, yielding a filtered frame elementhaving a plurality of associated filtered pixel values; means forselecting a first reference pixel from the filtered frame element, suchthat the first reference pixel has the highest value of the filteredpixel values; means for determining a first optic flow based on an opticflow equation applied at a location of the first reference pixel; meansfor performing pattern matching between the normalized pixel values of:the one normalized frame element of the first image frame; and thenormalized frame elements of the second image frame, being test frameelements, yielding a plurality of pattern matching scores, each of thetest frame elements having one of the pattern matching scores, whereinthe pattern matching is performed such that for each one of the testframe elements, a displacement vector between a location of the onenormalized frame element and a location of the one test frame element isrestricted by a region which is at least partially based on the firstoptic flow; and means for determining a motion vector at least partiallybased on the displacement vector associated with one of the test frameelements having a lowest one of the pattern matching scores.